Skip to content

feat: add OpenAI Agents SDK support to OpenInference mapper - #366

Open
liramon2 wants to merge 14 commits into
strands-agents:mainfrom
liramon2:openai-openinference
Open

feat: add OpenAI Agents SDK support to OpenInference mapper#366
liramon2 wants to merge 14 commits into
strands-agents:mainfrom
liramon2:openai-openinference

Conversation

@liramon2

@liramon2 liramon2 commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Description

Add OpenAI Agents SDK support to OpenInference mapper. This handles both Cloudwatch ADOT and in-memory exporter formats. No inheritance or composition is used because that is a one-way door decision that calls for a larger refactor.

This also fixes parent span ids in OpenInferenceSessionMapper so that they point to converted spans. This preserves the agent-tool scopes when converting from OpenAI Agent traces (or other OpenInference traces) to evaluator inputs.

Related Issues

#364

Documentation PR

Type of Change

New feature

Testing

How have you tested the change? Verify that the changes do not break functionality or introduce new warnings.

  • I ran hatch run prepare

Checklist

  • I have read the CONTRIBUTING document
  • I have reviewed and understand every line of code in this PR, including any generated by AI tools, and I can explain why it works
  • My change is focused and reasonably small; I have split unrelated work into separate PRs
  • I have added any necessary tests that prove my fix is effective or my feature works
  • I have updated the documentation accordingly
  • I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@github-actions github-actions Bot added enhancement New feature or request area-tracing Trace/session ingestion: providers, session mappers, extractors, telemetry/OTEL labels Aug 12, 2026
@liramon2
liramon2 deployed to auto-approve August 12, 2026 21:35 — with GitHub Actions Active
@liramon2

Copy link
Copy Markdown
Contributor Author

@strandly-the-agent Review this PR. Consolidate findings into a single comment.

@strandly-the-agent

This comment was marked as resolved.

@poshinchen

Copy link
Copy Markdown
Contributor

@strandly-the-agent Here are the instrumentation:

  1. "https://github.com/traceloop/openllmetry/tree/main/packages/opentelemetry-instrumentation-openai-agents",
  2. "https://github.com/traceloop/openllmetry/tree/main/packages/opentelemetry-instrumentation-openai",
  3. "https://github.com/Arize-ai/openinference/tree/main/python/instrumentation/openinference-instrumentation-openai-agents",
  4. "https://github.com/Arize-ai/openinference/tree/main/python/instrumentation/openinference-instrumentation-openai"

Here are the current findings:

For OpenAI agents SDK + traceloop instrumentation:

  • It mostly follows GenAI conventions
  • The instrumentation does not populate agent spans' available_tools . Tools only show up as descendant ToolExecution spans, so available_tools will be handled by backfilling from descendant tool spans
  • The traces have an empty root span that serves as a wrapper. Its operation is invoke_agent instead of the genai convention of invoke_workflow . These should be stripped to avoid having an empty agent span.
  • Agents that perform handoffs represent orchestrator and sub-agent spans as siblings. Need to use agent_handoff spans to correct parent ids if having valid parent ids is relevant

For OpenAI agents SDK + OpenInference instrumentation:

  • It mostly follows openinference conventions
  • input.value and output.value are not agent span attributes. They are instead on descendant LLM spans. So they are handled with a walk down to those spans and looking into llm.input_messages.N.message.content.0.message_content.text and llm.output_messages.N.message.contents.0.message_content.text
  • Available tools from _trace_tools_map are scoped to the trace while multiple agents can exist in the trace. This makes judges compare orchestrator tool calls against all available tools in the trace, causing false positives (bug seen last week). This can be handled by copying LLM tool schemas llm.tools.N.tool.json_schema onto AGENT spans and extracting them into available_tools . Granted, this can be deferred since the other frameworks using openinference also don't handle multi-agent traces correctly.

NOTE: In the current mappers' implementation strategy, I don't think we should use composition, and inheritance. And the mappers are grouped by instrumentation frameworks.

I need you to:

  1. verify the above claims,
  2. tell me how to implement the classes, should they be an additional class or just have the logics in the current mapper(s).
  3. I don't think we need inheritance for now. Tell me the suggestions

@strandly-the-agent

Copy link
Copy Markdown

Verified all seven claims against the instrumentation sources (openinference-instrumentation-openai-agents 2.0.0, opentelemetry-instrumentation-openai-agents 0.62.3 from Traceloop) and the captured fixtures in both PRs (#366 at bc6422d, #365 at ebc5c5a). Six hold; three need precision that changes the remedy.

# Claim Verdict
T1 Traceloop mostly follows GenAI conventions ✅ (deviations noted below)
T2 Traceloop doesn't populate available_tools → backfill from descendant tool spans ⚠️ true for the fixtures, but current v0.62.3 emits gen_ai.tool.definitions — prefer it; backfill has a semantic trap
T3 Empty root wrapper with invoke_agent → strip it ✅ (one precision: semconv has no invoke_workflow, and the wrapper's attrs change across versions)
T4 Handoff agents are siblings → reparent via agent_handoff spans ✅ (v0.62.3 adds a simpler signal: gen_ai.agent.handoff_parent on the sub-agent span)
O1 OpenInference mostly follows its conventions
O2 input/output live on descendant LLM spans → walk + read message paths ⚠️ mechanism right; the key paths as written are off, and index selection is the actual hard part
O3 _trace_tools_map is trace-scoped → multi-agent false positives; copy llm.tools.* onto AGENT spans; deferrable ✅ — but the copy alone doesn't close the bug you saw; the orchestrator has to survive too

Design answer (Q2/Q3): agree — no inheritance, and no new classes. openai_agents is a producer inside two convention families, not a framework of its own: keep its logic inside the two family mappers (OpenInferenceSessionMapper, GenericGenAISessionMapper) as scope-gated hooks in a fixed three-phase shape — normalize → shared convert → post-link. Applied consistently, that means folding #365's subclass back into GenericGenAISessionMapper. Details and a sketch below.

Traceloop verification (T1–T4)

T1 — GenAI conventions ✅. Fixture chat spans carry gen_ai.operation.name (chat/execute_tool/invoke_agent), the new semconv message format (gen_ai.input.messages/gen_ai.output.messages as role/parts JSON), gen_ai.usage.*, gen_ai.agent.name/id/description. Deviations to plan for: per-turn wrapper spans with gen_ai.operation.name: "unknown" (8–12 attrs, no content); the custom agent_handoff operation (not in semconv); deprecated gen_ai.system alongside gen_ai.provider.name.

T2 — ⚠️ version-dependent, and the backfill has a semantic trap.

  • In both feat(mappers): add OpenAI Agents support to GenAI session mapper #365 fixtures the claim is exactly right: the only tool-ish attributes anywhere are gen_ai.tool.call.{name,arguments,result,type} on execute_tool spans — zero declared-tool schemas.
  • But the current release emits the real declared list: v0.62.3's _end_generation_span sets gen_ai.tool.definitions (full name/description/parameters JSON, via _extract_tool_definitions) on generation/response spans when content tracing is on (_hooks.py:1021-1028; runs for GenerationSpanData/ResponseSpanData, :800-803). The feat(mappers): add OpenAI Agents support to GenAI session mapper #365 fixtures simply predate it.
  • So the remedy should be: prefer gen_ai.tool.definitions when present, fall back to backfill. Two reasons the backfill alone is worth improving: (i) descendant ToolExecutionSpans are tools used, not tools available — a tool-selection judge sees a shrunken candidate set with the un-chosen options missing, which is the mirror image of the false-positive bug you saw on the OpenInference side (candidate set too big there, too small here); (ii) feat(mappers): add OpenAI Agents support to GenAI session mapper #365's backfill builds ToolConfig(name=...) only — no description/parameters for the judge to reason over, while gen_ai.tool.definitions carries both.

T3 — ✅ confirmed in both fixtures. Root Agent workflow span with gen_ai.operation.name: "invoke_agent" and no agent name / no messages (live: 5 attrs, all provider/server boilerplate). Stripping is right, and #365's rule (no gen_ai.agent.name and no gen_ai.input.messages → skip) is the robust one — better than keying on the operation name, because the wrapper's attributes are version-dependent: in v0.62.3 the root gets traceloop.span.kind: workflow and no gen_ai.operation.name at all (on_trace_start, _hooks.py:676-688). One nit on the claim's wording: GenAI semconv defines no invoke_workflow operation (chat, execute_tool, invoke_agent, create_agent, embeddings, …) — workflow-ness is Traceloop's own traceloop.span.kind, so there's no "correct" operation the wrapper should have had; it just needs stripping.

T4 — ✅ confirmed. In the ADOT fixture, invoke_agent coordinator and invoke_agent math_specialist are siblings (children of the same turn wrapper under the root), with the agent_handoff span (gen_ai.handoff.from_agent: coordinator, gen_ai.handoff.to_agent: math_specialist) sitting under the coordinator's turn — the name-based reparenting in #365's _apply_handoff_reparenting is the right join, and the ebc5c5a tiebreaker doc covers the duplicate-name case. Two additions:

  • The agents-as-tools pattern needs no correction — in the live fixture, execute_tool ask_math_specialist → nested invoke_agent math_specialist nests properly already. Sibling-flattening is handoff-specific.
  • v0.62.3 also stamps gen_ai.agent.handoff_parent directly on the handed-off agent's own invoke_agent span (utils.py:14, applied in _start_agent_span via _reverse_handoffs_dict). When present it's a simpler, collision-free signal than the handoff-span join — worth preferring, with the span-join as fallback for older captures like the fixture.
OpenInference verification (O1–O3)

O2 — mechanism ✅, paths and indices need precision. Verified against the v2.0.0 instrumentor source and both #366 fixtures:

  • Input: plain-text messages carry the scalar llm.input_messages.N.message.content — that's the only input shape in both shipped fixtures. The contents.0.message_content.text shape appears for structured/multimodal input (_get_attributes_from_message_param emits one or the other, not both). The path as written in your comment (...message.content.0.message_content.text) mixes the two — a mapper reading only that would drop every plain-text trace. Read scalar first, contents.0...text as fallback (that's also exactly review finding 1c: the current PR code reads only the scalar and drops multimodal spans).
  • Output: contents.0.message_content.text on v2.0.0, scalar .message.content on v1.6.1 — both fallbacks needed (the PR does this part right).
  • The hard part is N, not the path: the Responses-API path numbers input from 1 (_get_attributes_from_input, msg_idx=1), the chat-completions path (GenerationSpanData — LiteLLM/Azure/compatible endpoints) from 0, and reasoning items consume output indices. So: highest-index message with role == "user" for input, highest-index non-reasoning text for output — hardcoded indices break multi-turn, reasoning models, and the chat-completions path (review finding 1, with repros).

O3 — ✅ verified, with one important limit. _trace_tools_map[trace_id] is populated from every LLM span in the trace during conversion, so in a multi-agent trace each agent's available_tools is the trace-wide union — your false-positive mechanism is exactly what my LLM-context pass reproduced (a judge asked to justify handoff to math_specialist({}) against a candidate set that doesn't contain it). Copying llm.tools.N.tool.json_schema onto AGENT spans fixes the scoping, but as shipped it doesn't close the bug you saw, for two reasons:

  • The orchestrator's AGENT span is currently dropped (its final LLM turn is tool-calls-only, so output.value never gets injected and the _is_agent_invocation_span gate rejects it). Its handoff tool span is then misattributed to the surviving specialist by the orphan fallback (types/trace.py:202-206) and judged against the specialist's tools — same species of false positive, one hop over. The fix needs the orchestrator to survive (or orphaned tool spans to stay unowned), not just the tool copy. That's review blocker 2 / Question 1 — it's the decision this hinges on.
  • The copy takes schemas from the earliest LLM span only, so per-turn tool filtering (FunctionTool.is_enabled, MCP without cache) loses entries. Deriving indices from keys and merging turns is cheap.
  • The "deferrable because other openinference producers don't handle multi-agent either" part is accurate: langchain dedups to one agent span (is_langchain gate), and smolagents/claude multi-agent traces get the same trace-wide union today.
Design recommendation (Q2/Q3): no new classes, no inheritance — three named phases inside the two family mappers

The rule that matches the repo you already have: one mapper per emitting-convention family (OpenInferenceSessionMapper already serves langchain/smolagents/claude via scope-gated tweaks; GenericGenAISessionMapper serves GenAI-convention traces), routed by detect_otel_mapper on scope. openai_agents is a producer inside two families — OpenInference scope in #366, GenAI scope in #365 — so it doesn't earn a class in either. A per-producer class would duplicate each family's conversion machinery (span-kind detection, message parsing, ADOT handling) for an 80-line delta.

Why I'd agree about inheritance specifically: #365's subclass works by overriding _convert_trace / _convert_agent_invocation_span — it's coupled to GenericGenAISessionMapper calling those exact template methods in that exact order. A refactor of the base silently changes the subclass's behaviour with no test failing in the subclass's own file (fragile-base-class). Scope-gated hooks keep the coupling visible in one place. Composition (a strategy object per producer) would also work but buys nothing over a plain method-dispatch table at this size — it's structure without payoff.

The shape I'd standardize (both mappers already approximate it):

  1. Normalize (per-producer, before shared conversion): rewrite raw span dicts into the family's canonical shape. _normalize_smolagents_span and _normalize_openai_agents_trace are this today. Registered in a dispatch table rather than a growing if-chain:
    # producer scope -> (granularity, hook)
    _PRODUCER_NORMALIZERS = {
        SCOPE_OPENINFERENCE_SMOLAGENTS: ("span", _normalize_smolagents_span),
        SCOPE_OPENINFERENCE_OPENAI_AGENTS: ("trace", _normalize_openai_agents_trace),
    }
  2. Convert (shared, producer-blind): one code path per family. Producer conditionals should not appear here — if one is needed, the data should have been normalized instead.
  3. Post-link (per-producer where needed, after conversion): structural fixes on converted spans — feat(mappers): add OpenAI Agents support to GenAI session mapper #365's handoff reparenting and tools backfill live here; bridge_parent_gaps is the family-agnostic version and stays shared in utils.

Three guardrails, all from bugs already found in these two PRs: each hook is exception-isolated (one producer's quirk must degrade to a warned-and-skipped trace, not kill the session — the #366 normalizer currently runs outside every try/except); each hook checks scope on the spans it touches (the #366 walk doesn't); and each hook gets producer fixtures that pin it (mutation testing showed the bridge_parent_gaps call can be deleted with all 385 tests green).

Applied to the two PRs:

  • feat: add OpenAI Agents SDK support to OpenInference mapper #366: keep everything in OpenInferenceSessionMapper — no new class. The changes it needs are the review fixes plus, optionally, the dispatch table above.
  • feat(mappers): add OpenAI Agents support to GenAI session mapper #365: the consistent no-inheritance answer is to fold OpenAIAgentsOtelSessionMapper into GenericGenAISessionMapper: wrapper-strip becomes a normalize/skip rule, handoff-reparent + tools-backfill become a post-link step, both gated on SCOPE_OPENAI_AGENTS, and detect_otel_mapper:140-141 routes to the generic mapper. It's ~118 lines whose three behaviours are already discrete methods, so folding is cheapest now, before it grows. Keeping the subclass is workable, but then the codebase carries two different strategies for the same problem — which is the divergence that prompted your question.

One thing I'd not do: don't introduce a producer-level abstraction (base class or protocol for "producers") until a third family needs it. Two call sites is a pattern; codifying it early is how the mapper layer gets rigid.

Verification by strandly-the-agent: claims checked against instrumentor sources (openinference 2.0.0, traceloop 0.62.3) and both PRs' fixtures; OpenInference items re-verified with the repros from my review above. As ever — my analysis to check, your call to make.

@liramon2

Copy link
Copy Markdown
Contributor Author

@strandly-the-agent Review this PR. Consolidate findings into a single comment. For your questions:

  • The coordinator should exist, but the workflow wrapper should not
  • They can land in any order. The parallel fix was a coincidence
  • No live integration tests yet

@strandly-the-agent

This comment was marked as resolved.

@liramon2

Copy link
Copy Markdown
Contributor Author

@strandly-the-agent Review this PR. Consolidate findings into a single comment.

@strandly-the-agent

This comment was marked as resolved.

@liramon2
liramon2 force-pushed the openai-openinference branch from dbbbd53 to ccc3086 Compare August 19, 2026 15:04
@liramon2
liramon2 deployed to auto-approve August 19, 2026 15:07 — with GitHub Actions Active
@liramon2

Copy link
Copy Markdown
Contributor Author

@strandly-the-agent Review this PR (resolved merge conflicts + added test since last review). Consolidate findings into 1 comment.

@strandly-the-agent

Copy link
Copy Markdown

Re-reviewed at 7de7eb9 (rebase onto dce1b8a + ccc3086 “capture result from bridge_parent_gaps” + 7de7eb9 the bridge test). Self-run follow-up; everything below executed at this head.

All clear — the round-4 🟡 is closed, the rebase is drift-free, and I have no findings.

Follow-up review by strandly-the-agent — rebase + fixes-only delta, self-run per protocol: accumulated-diff comparison for rebase drift, both mutations re-run, suites green at the new base. Nothing outstanding from my side across all five rounds.

- openinference-instrumentation-langchain (LangChain / LangGraph)
- openinference-instrumentation-smolagents (HuggingFace smolagents)
- openinference-instrumentation-claude-agent-sdk (Claude Agent SDK)
- openinference-instrumentation-openai-agents (OpenAI Agents SDK)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how do we pick up this naming? Wondering claude-agent-sdk but for openai agent sdk we only do openai-agents

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are the official package names from the OpenInference repository: https://github.com/Arize-ai/openinference/tree/main

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

return llm_spans

@staticmethod
def _get_message_indices(attrs: dict, prefix: str) -> list[int]:

@JackYPCOnline JackYPCOnline Aug 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: overall, can we make function naming more accurate

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 8bf879e. Standardized the verbs from get to extract/collect.

indices.add(int(seg))
return sorted(indices, reverse=True)

@classmethod

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these helpers need class-level invocation or subclass override behavior? Origin code is just some private helpers.
if they are a must, can we make code more consistent?

@liramon2 liramon2 Aug 19, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They don't need access to self, but then again other private helpers in the file don't access self. So to answer your question, they can just be private helpers. Done in 8bf879e

@liramon2

Copy link
Copy Markdown
Contributor Author

@strandly-the-agent Review this PR. Consolidate any findings into 1 comment.

@strandly-the-agent

Copy link
Copy Markdown

Re-reviewed at 02a76ed (delta since 7de7eb9: formatting · cb15d94 foreign-scope checks · 8bf879e helper rename/reorg · 02a76ed “remove stale error check”). Self-run follow-up; everything below executed at this head.

One 🟡 to flag: the “stale” error check wasn't stale. 02a76ed removes ValueError from the except tuple in _extract_tools_from_attributes — but it was catching pydantic's ValidationError (a ValueError subclass), and removing it resurrects a failure mode this PR deliberately fixed in round 1. Everything else in the delta checks out clean.

Commit Verdict
cb15d94 foreign-scope checks ✅ verified — closes the long-standing mixed-scope appendix item, both directions
8bf879e rename/reorg ✅ behaviour-neutral — span-tuple diff across all 7 fixtures vs 7de7eb9: 0 changes; helper bodies moved verbatim
02a76ed remove ValueError 🟡 regression — A/B repro below; one-word restore
Suites ✅ mapper 399 passed · extractors/types/detectors 368 passed
🟡 the removed ValueError was catching pydantic ValidationError — a bad tool schema now drops the whole span again (A/B verified)

ToolConfig.name is a required str (types/trace.py:52), so a tool schema without a usable name — e.g. an OpenAI envelope whose inner function dict lacks one, or "name": 123 — makes ToolConfig(name=None, …) raise pydantic.ValidationError, which is a ValueError subclass. That's what the removed clause was catching (the TypeError scalar-schema case it once also shielded was properly fixed in the per-span normalizer two rounds ago — that half really was stale; this half wasn't).

Same input, previous head vs this head — an AGENT span carrying one nameless schema and one good one:

at 7de7eb9:  AgentInvocationSpan count = 1   tools=['good_tool']        # bad tool skipped
at 02a76ed:  AgentInvocationSpan count = 0   (WHOLE SPAN DROPPED)
             "Failed to convert span ag: 1 validation error for ToolConfig … name … input_value=None"

The exception escapes to _build_trace's broad per-span except Exception, so one malformed tool schema silently discards the entire agent invocation — including the valid tools next to it. This is byte-for-byte the round-1 finding that motivated adding ValueError (my round-1 comment noted it was not redundant for exactly this case; the round-2 probes confirmed “that's the ValueError catch working”).

Reachability is contrived-data territory (real instrumentors emit named tools), so 🟡 not 🔴 — but it's a silent regression of this PR's own earlier behaviour. Fix: restore ValueError to the tuple, or skip schemas where not isinstance(tool_info.get("name"), str) before constructing ToolConfig. A pin test is cheap: the repro above as a unit test asserting tools == ['good_tool'] and span survival.

cb15d94 — foreign-scope checks verified in both directions
  • A smolagents-scoped AGENT span that is an ancestor of openai_agents CHAIN/LLM spans in a shared trace is no longer normalized (attrs untouched after map_to_session — previously it got openai content injected).
  • A foreign-scope LLM descendant (e.g. langchain) is no longer harvested for an openai agent's prompt/response — an openai AGENT with only foreign LLM descendants is now dropped rather than borrowing foreign content.
  • The new test_foreign_scope_llm_not_used_for_agent_normalization pins the second direction. Both openai fixtures: span-tuple diff vs 7de7eb9 = 0 changes.

Follow-up review by strandly-the-agent — self-run per protocol: A/B'd the delta against the previous head, span-tuple-diffed all 7 fixtures, re-ran the suites. The 🟡 is a one-word restore plus a small pin test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-tracing Trace/session ingestion: providers, session mappers, extractors, telemetry/OTEL enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants